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Abstract 


Cognitive  diagnosis  of  expertise  relies  on  characterizing  expertise  in  the 
domain  of  interest.  The  focus  of  this  project  was  on  characterizing  and 
assessing  design  problem  solving  in  the  area  of  digital  circuit  design.  A 
combination  of  think-aloud  protocols  and  computer  traces  of  subject 
problem-solving  behavior  was  used  to  elucidate  the  cognitive  processes 
involved  in  designing  combinational  and  complex  sequential  circuits.  The 
project  had  four  components:  (i)  study  the  differences  between  experts  and 
intermediates  in  solving  combinational  circuit  problems,  (ii)  characterize 
planning  behavior  and  its  impact  on  the  quality  of  solutions  for 
combinational  circuits,  (iii)  validate  the  effectiveness  of  traces  collected  by  a 
design  tool  in  assessing  problem-solving  behavior  for  combinational  circuits, 
and  (iv)  characterize  and  assess  problem-solving  behavior  for  complex 
sequential  circuits.  The  combinational  circuit  studies  revealed  local  planning 
in  problem  solving  but  little  global  planning,  mainly  because  of  the  routine 
nature  of  the  design  process.  Unlike  other  domains  reported  in  the  literature, 
clear  differences  between  intermediate  and  expert  problem  solving  did  not 
emerge.  On  the  other  hand,  the  complex  sequential  circuit  design  problem 
revealed  significant  differences  between  expert,  intermediate,  and  novice 
problem  solvers.  Problem  solving  was  successfully  modeled  by  integrating 
three  cognitive  models  of  human  design:  (i)  problem  decomposition,  (ii) 
transformation  and  iterative  refinement,  and  (iii)  analogy  /  prototype 
models. 
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Assessing  Digital  Circuit  Design 
ONR  Grant  N00014-91-J-1780 

The  focus  of  this  project  was  on  the  assessment  of  design  problem 
solving  in  the  area  of  digital  circuit  design.  Think-aloud  protocol 
methodology  was  employed  to  elucidate  the  cognitive  processes  involved  in 
executing  the  design  of  these  kinds  of  complex  circuits.  However,  due  to  the 
labor-intensive  nature  of  protocol  analysis,  we  were  interested  in  ascertaining 
the  degree  to  which  computer-based  design  tools  would  provide  sufficient 
information  to  accomplish  classificatory  assessment  of  design  expertise. 

Previous  research  on  the  study  of  design  processes  suggested  that 
although  it  is  a  highly  constructive  process,  there  are  some  general  principles 
and  heuristics  that  apply  (e.g.,  Garrod  &  Borns,  1991).  For  example,  Goel  and 
Pirolli  (1989)  studied  design  tasks  in  different  domains;  architecture, 
mechanical  engineering,  and  instructional  design  and  collected  a  total  of  12 
think-aloud  protocols.  From  this  database,  the  protocols  of  three  participants, 
one  from  each  domain,  were  discussed.  The  three  participants  varied  in 
experience:  ten  years'  design  experience  in  designing  industrial  training 
material,  six  years  professional  experience  as  an  architect,  and  limited  design 
experience  in  mechanical  engineering. 

From  the  protocol  analysis,  Goel  and  Pirolli  identified  eight  significant 
invariants  in  the  problem  spaces  of  the  participants  in  the  three  different 
design  disciplines: 

•  extensive  problem  structuring, 

•  extensive  performance  modeling, 

•  personalized  or  institutionalized  evaluation  functions  and  stopping 
rules, 

•  a  limited  commitment  mode  control  strategy, 

•  constraint  propagation, 

•  the  role  of  abstractions  in  the  transformation  of  goals, 

•  the  use  of  artificial  symbol  systems, 

•  solution  decomposition  into  leaky  modules.i 

These  invariants  are  generally  consistent  with  the  results  of  other  researchers 
and  describe  design  behaviors  that  are  consistent  across  different  domains. 


1  A  leaky  module  is  a  nonindependent  module.  A  decision  made  in 
one  leaky  module  could  have  consequences  in  several  others. 
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Designing  Simple  Digital  Circuits 

We  began  our  investigation  of  design  in  digital  circuits  by  examining 
the  solution  process  for  simple  digital  circuits,  i.e.,  combinational  logic 
circuits.  Our  primary  purpose  in  this  study  was  to  characterize  the 
performance  of  students  just  beginning  to  study  the  design  and  circuit  layout 
process  (intermediates)  and  compare  it  to  that  of  solvers  who  were  advanced 
graduate  students  or  had  experience  working  in  the  field  (experts).  Although 
there  exist  standard  procedures  for  designing  combinational  logic  circuits, 
gaining  proficiency  in  their  design  demands  that  the  designer  is  aware  of 
trade-offs  between  different  performance  parameters.  T)rpical  circuit 
performance  parameters  include  minimizing  the  total  number  of  gates  used, 
the  input-output  delay,  and  total  power  consumption.  As  a  result,  a  number 
of  alternative  design  solutions  may  be  possible,  each  of  which  satisfies  the 
functional  requirements,  but  only  a  small  number  (one  or  more)  may  be 
considered  suitable  when  taking  into  account  all  the  different  requirements. 
Furthermore,  the  complexity  of  most  design  problems  requires  that  they  be 
broken  down  into  subproblems,  each  of  which  can  then  be  solved 
independently.  However,  the  decomposition  process  is  often  difficult  and 
improper  decompositions  often  create  subproblems  that  interact  thus 
complicating  their  solution  process.  In  such  situations,  designers  are  often 
found  to  produce  incorrect  or  suboptimal  solutions. 

The  foregoing  characteristics  of  the  domain  served  as  an  initial  set  on 
which  we  wished  to  contrast  expert-  and  intermediate-level  designers.  The 
results  of  that  work  are  described  in  James,  Goldman,  &  Vandermolen  (1994) 
and  are  only  summarized  here. 

A  total  of  seven  participants,  two  experts  and  five  intermediates, 
designed  a  1-bit  full  subtractor  and  were  given  two  criteria  for  a  "good" 
design.  The  first  was  to  minimize  cost  (i.e.,  minimize  the  number  of  gates 
used),  and  the  second  was  testability  of  the  circuit.  The  standard,  or  textbook, 
procedure  for  solving  this  problem  relies  on  a  sequence  of  six  components. 
They  are: 

•  imderstanding  the  problem  to  determine  and  list  the  input  and 
output  variables,  and  the  nature  of  their  relations,  from  a  word 
description  of  the  problem, 

•  completing  the  truth  table  to  express  and  make  explicit  the 
complete  relations  between  input  and  output  variables, 

•  constructing  Karnaugh  maps  (K-maps)  to  generate  minimal 
Boolean  expressions  that  express  the  relation  between  individual 
output  variables  and  the  input  variables, 

•  expressing  the  results  of  the  K-map  form  as  Boolean  expressions, 

•  implementing  these  Boolean  expressions  as  combinational 
circuits,  and 

•  evaluating  the  circuits  generated  for  correctness  and  optimality. 
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Think-aloud  verbal  protocols,  paper  and  pencil  solutions  that  participants 
generated  when  they  solved  the  problem,  and  retrospective  interviews 
constituted  the  raw  data. 

Based  on  previous  expert-novice  contrastive  research,  we  expected  that 
the  more  expert  participants  would  exhibit  more  planning  behavior  than  the 
less  expert  participants  (e.g.,  Larkin,  1980),  have  stronger  self-monitoring 
skills  (e.g.,  Chi,  1978;  Chi,  1987;  Chi,  Glaser,  &  Rees,  1982;  Larkin,  1983;  Miyake 
&  Norman,  1979;  and  Simon  &  Simon,  1978),  and  consider  alternative 
solutions  in  attempting  to  achieve  an  optimal  solution  (Akin,  1977).  The 
combination  of  these  should  lead  to  the  more  expert  problem  solvers  making 
attempts  to  optimize  their  solution,  and  also  engage  in  more  verification  and 
evaluation,  thus  producing  "better"  solutions  overall.  We  expected  that  the 
less  skilled  participants  would  exhibit  little  planning  in  the  beginning  of  their 
solution  process,  attempt  to  apply  the  standard  solution  methodologies  in  a 
brute-force  manner,  and  stop  to  decide  how  to  proceed  when  they  hit  an 
impasse  (e.g.,  Paige  &  Simon,  1966).  Therefore,  even  if  they  generated  correct 
solutions,  more  likely  than  not,  their  solutions  would  be  suboptimal. 

James  et  al.  (1994)  differentiated  between  the  type  and  location  of 
planning  in  the  problem  solution.  Global  planning  was  defined  as  planning 
that  concerned  not  just  the  component  step  in  which  it  occurred,  but 
situations  in  which  the  designer  was  considering  the  impact  of  a  sequence  of 
component  steps  on  the  solution  process,  or  the  impact  of  one  component 
step  on  another.  Local  planning  dealt  only  with  the  solution  component  in 
which  it  occurred. 

James  et  al.  found  that  the  two  experts  generally  proceeded  without 
difficulty,  but  they  failed  to  correct  errors  in  their  solutions,  and  only  one  of 
them  engaged  in  global  planning  prior  to  beginning  the  problem  solution. 
The  latter  two  results  were  tmexpected  given  the  previous  characterizations 
of  expert  performance  (e.g.,  Chi  et  al.,  1988;  Larkin,  1980).  However,  as  has 
been  reported  in  other  domains,  James  et  al.  fovmd  that  all  of  the 
intermediates  were  characterized  by  an  overall  strategy  of  directly  attacking 
problem  understanding  by  attempting  the  first  step  in  the  solution  algorithm. 
For  all  of  the  participants,  evaluation  largely  consisted  of  verifying  the  last 
step  (as  opposed  to  checking  if  the  solution  matched  the  functional 
specifications  stated  in  the  word  problem)  and  this  did  not  lead  to  their 
detecting  errors  and  suboptimal  solutions.  In  addition,  only  one  of  the 
intermediates  engaged  in  any  global  planning  and  it  occurred  not  at  the 
beginning  of  the  solution  but  in  the  middle.  Localized  planning  occurred 
fairly  frequently,  especially  among  the  intermediates,  and  typically  occurred 
in  the  middle  of  the  solution  process.  In  general,  the  intermediates  in  this 
study  did  not  formulate  global  plans,  but  did  exhibit  local  plaiming 
throughout  their  solutions. 

The  James  et  al.  study  provides  some  information  about  the  processes 
of  solvers  at  various  levels  of  expertise  in  a  previously  unexplored  domain. 
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However,  the  James  et  al.  study  appeared  to  show  considerable  variability  in 
performance  and  in  the  planning  processes.  The  high  variability  and  the 
small  number  of  participants  makes  generalizing  the  results  difficult.  The 
results  of  James  et  al.  (1994)  indicate  that  intermediates  tend  to  implement  the 
standardized  circuit  design  process  mechanically,  and  do  not  reflect  on  the 
consequences  and  results  generated  in  one  step  on  subsequent  steps.  This  was 
clear  from  their  solution  traces  in  that  (a)  they  did  not  attempt  to  compare  the 
results  at  subsequent  steps  with  information  at  previous  steps  for  verification 
purposes,  and  (b)  they  failed  to  look  for  common  terms  in  the  two  output 
equations  so  that  the  number  of  gates  could  be  reduced  in  the 
implementation. 

In  the  expert  group,  one  of  the  experts  formulated  a  detailed  global  plan 
prior  to  beginning  problem  solving,  while  the  other  expert  did  not.  This 
finding  does  not  necessarily  contradict  the  literature  on  expert  behavior.  The 
expert  who  failed  to  verbalize  a  global  plan  appeared  to  have  been  carrying 
out  a  standard  set  of  solution  procedures  that  may  have  been  automated  and, 
therefore,  not  accessible  to  verbalization.  These  data  suggest  the  possibility 
that  if  the  intermediates  started  thinking  about  the  steps  in  the  solution  at  the 
beginning  of  the  problem  solving  process,  they  might  be  more  likely  to 
proceed  through  the  steps  in  the  algorithm  more  smoothly  and  direct  their 
attention  to  verification  and  optimization  issues  they  did  not  otherwise 
consider. 

A  follow-up  study  to  the  "subtractor"  combinational  circuit  design 
study  was  conducted  to  pursue  the  plarming  issue.  A  second  purpose  was  to 
validate  the  computer-based  design  tool  that  had  been  developed 
simultaneously  with  conducting  the  think-aloud  protocol  study.  Planning 
and  its  relationship  to  expertise  in  solving  design  problems  were  the  purview 
of  a  Master's  Thesis  conducted  by  Carol5m  M.  James.  Data  for  the  Design 
Assistant  validation  study  were  generated  in  the  course  of  collecting  the 
James  Thesis  data. 

Design  of  the  Planning  and  Validation  Study 

Twenty  participants  designed  two  combinational  circuits  using  the 
Design  Assistant  tool.  Participants,  imdergraduate  or  first  year  graduate 
electrical  engineering  students,  were  paid  volunteers.  The  majority  of 
students  were  enrolled  in  a  senior-level  digital  design  course.  They  had  all 
completed  one  introductory  course  in  digital  circuit  design  and  had  very 
limited  design  experience.  Each  participant  was  asked  to  solve  two  problems. 
Problem  A  was  the  Subtractor  problem  used  in  our  first  verbal  protocol  study 
Games  et  al.,  1994).  Problem  B  was  the  Job  Skill  problem,  developed  for  the 
present  study  to  be  comparable  to  Problem  A  in  terms  of  the  processes 
necessary  for  designing  the  circuit. 

Problem  A  required  the  participants  to  design  a  one-bit  full  subtractor, 
taking  into  accoimt  several  design  constraints  (see  Table  1).  The  participants 
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were  provided  with  a  written  description  of  the  desired  function  of  the  circuit 
(subtraction)  rather  than  being  explicitly  told  the  desired  input-output 
behavior.  The  problem  is  similar  to  an  example  often  used  in  class  (the  adder 
circuit),  but  different  enough  so  that  none  of  the  participants  should  have 
been  able  to  generate  the  specific  solution  for  this  problem  from  memory.  The 
participants  each  should  have  had  enough  background  knowledge  to  call  to 
mind  the  standard  framework  (textbook  strategy)  for  solving  this  type  of 
problem.  The  circuit  from  Problem  A,  the  Subtractor  problem,  requires  three 
inputs  and  two  outputs. 

Problem  B,  the  Job  Skill  problem,  required  the  participants  to  solve  a 
problem  in  which  the  input-output  relations  are  described  in  words,  rather 
than  a  description  of  the  function  of  the  circuit  as  in  the  Subtractor  problem 
(see  Table  1).  Both  problems  are  combinational  logic  problems  and  thus,  the 
solution  process  for  both  follows  the  same  standard  algorithm  (textbook 
strategy).  The  circuit  from  the  Job  Skill  problem  required  four  inputs  and  two 
outputs.  Both  problems  were  judged  by  the  instructor  of  the  design  course  to 
be  approximately  equal  in  difficulty  and  complexity. 

The  participants  solved  both  problems  using  the  Design  Assistant  tool. 
They  were  asked  to  think  aloud  during  the  solution  process  (c.f.,  Ericsson  & 
Smith,  1991).  Sessions  were  videotaped,  with  the  camera  focused  on  the 
computer  screen.  Audible  comments  by  the  participants  were  thus  correlated 
to  particular  actions  on  the  computer.  The  Design  Assistant  software  created  a 
trace  of  the  actions  taken  by  the  participant  but  could  not  trace  the 
participants'  verbalizations. 

During  the  first  of  two  sessions,  each  participant  worked  on  a  practice 
problem  to  familiarize  them  with  the  software,  and  then  solved  their  first 
problem.  The  second  problem  was  administered  during  a  separate  session  in 
which  explicit  planning  instructions  were  given  to  each  participant.  Each 
participant  solved  all  of  the  problems  independently  without  time 
constraints.  The  participants  were  allowed  to  use  reference  materials  (e.g., 
textbooks)  which  were  provided  if  necessary.  The  average  time  for  solving 
each  problem  was  two  hours.  Each  participant  was  alone  during  problem 
solving,  but  an  experimenter  was  within  hearing  distance  to  answer  any 
questions  or  to  prompt  the  participant  to  continue  talking  during  silent 
periods. 

After  each  participant  solved  the  second  problem,  they  were  debriefed 
and  had  an  opportunity  to  comment  on  the  study  and  to  make  suggestions 
for  future  improvements  to  the  software  they  had  been  using  during  the 
study. 

Planning  was  manipulated  by  instructions  prior  to  solving  the  second 
problem.  (Problems  A  and  B  served  as  "first"  or  "second"  problems  equally 
often  across  participants.)  Before  solving  the  second  problem  each  participant 
was  asked  to  formulate  an  overall  plan  for  solving  the  problem  and  was 
given  an  example  of  an  overall  plan  (see  Table  2).  The  example  plan 
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illustrated  that  generating  an  overall  plan  meant  laying  out  things  that  need 
to  be  thought  about,  how  to  accomplish  them,  and  what  contingencies  need 
to  be  considered. 

Validation  of  the  Design  Assistant  Tool 

The  methodology  for  validating  the  Design  Assistant  tool  involved 
comparing  the  characterization  of  problem  solving  obtained  by  examining  the 
computer-generated  solution  traces  with  that  derived  from  analysis  of  the 
videotapes  of  the  problem  solving.  Problem  solution  was  characterized  by 

•  overall  solution  strategies  used, 

•  whether  the  Truth  table,  K-map  and  Boolean  components  were 
applied  to  the  output  variables  sequentially  (one  by  one)  or 
concomitantly  (parallel), 

•  whether  the  outputs  were  implemented  independent  of  one 
another  or  together  as  one  imit,  and 

•  correctness  of  the  final  solution. 

Coding  and  Analyses 

Four  measures  were  derived  from  the  flow  diagrams  for  each  problem 
solved  by  each  participant.  (Example  flow  diagrams  appear  in  Figures  1  and  2.) 
These  were  overall  solution  strategies,  intermediate  steps  strategies, 
implementation  strategies,  and  completeness. 

Overall  Solution  Strategies.  Three  types  of  Overall  Strategies  were 
distinguished  based  on  the  component  sequence  that  was  executed  by  the 
participants.  The  first  strategy  was  the  Textbook  Strategy  and  is  indicated  by  a 
linear  progression  through  the  Truth  Table,  K-maps,  Boolean  equations, 
implementation  and  evaluation.  The  second  was  the  Textbook  Strategy 
without  Boolean  equations,  in  which  no  Boolean  equations  were  written  for 
either  output.  The  third  was  the  Textbook  Strategy  without  K-maps.  In  this 
strategy,  either  K-maps  were  not  used  at  all  or  they  were  only  partially 
attempted  and  not  used  properly. 

Intermediate  Step  Strategies.  The  Intermediate  Step  Strategies  of  the 
solution  were  coded  as  either  Serial  or  Parallel.  A  solution  with  Serial  steps  is 
one  in  which  the  K-map,  grouping,  and  Boolean  expressions  were  solved  for 
one  output  before  the  other  (see  the  middle  section  of  Figure  1).  A  solution 
with  Parallel  steps  is  one  in  which  the  two  outputs  were  worked  on  in 
parallel:  K-map  for  first;  K-map  for  second;  Grouping  for  first,  etc.  (see  the 
middle  section  of  Figure  2). 

Implementation  Strategies.  The  Implementation  Strategies  were  coded 
similarly  to  the  Intermediate  Step  Strategies.  In  a  Parallel  Implementation 
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Strategy,  both  outputs  (e.g..  Result  and  B-out  for  the  Subtractor)  were 
implemented  and  then  simulated.  In  a  Serial  Implementation  Strategy,  one 
output  was  implemented  before  the  other. 

Completeness.  Intermediate  steps  and  implementation  could  be 
completed  for  only  one  of  the  two  inputs  or  for  both.  If  only  one  was 
implemented,  the  solution  was  less  complete  than  if  both  were  implemented. 

Two  raters  were  used.  One  rater  coded  each  participant's  two  problems 
from  the  videotapes  and  transcripts  made  from  them.  The  other  rater 
independently  coded  each  participant's  two  problems  from  the  computer¬ 
generated  trace  of  the  solution. 

Results  of  the  Analyses 

Overall  Solution  Strategies.  There  were  two  disagreements  between 
raters  on  the  20  problem  A  traces  coded  and  two  on  the  18  problem  B  traces 
that  could  be  coded.  (Two  problem  B  computer  traces  were  lost  due  to  a 
software  malfunction.)  The  distribution  based  on  the  coding  done  from  the 
videotape  is  provided  in  Table  3.  This  distribution  shows  that  the  majority  of 
solutions  followed  the  standard  textbook  algorithm.  The  next  most  frequent 
approach  was  to  leave  out  the  Boolean  component  and  work  from  the  K-map 
grouping  to  an  implementation.  The  disagreements  between  raters  concerned 
whether  or  not  K-map  or  Boolean  components  had  been  included.  For  the 
four  disagreements,  the  computer  trace  indicated  that  these  components  had 
been  attempted,  whereas  the  videotapes  showed  that  the  participants  had  not 
actually  worked  on  the  K-map  or  Boolean  components.  Disagreements  of  this 
type  were  due  to  the  fact  that  a  participant  could  have  opened  a  Boolean 
window  but  not  actually  worked  on  a  Boolean  expression.  The  computer  trace 
indicated  that  the  Boolean  window  had  been  opened  but  not  what  had  been 
done  in  that  window,  if  anything.  The  computer-trace  rater  assumed  that  if 
the  window  had  been  opened  work  on  K-maps  or  Boolean  expressions  had 
been  done.  In  contrast,  the  video  showed  whether  or  not  any  work  had  been 
done  in  these  windows. 

Intermediate  Steps  Strategies.  There  were  three  disagreements  on 
Problem  A  and  two  on  problem  B.  The  videotape  coding  showed  that  12  of 
the  problem  A  and  15  of  the  problem  B  solutions  used  a  parallel  strategy  and 
the  remainder  were  serial.  Disagreements  between  coders  were  not 
systematic,  i.e.,  one  rater  did  not  favor  parallel  versus  serial  designations.  The 
differences  between  raters  were  related  to  the  fact  that  the  software  did  not 
require  a  new  window  for  the  two  outputs.  It  was  not  possible  to  tell  from  the 
computer  trace  whether  the  student  was  working  on  one  or  two  outputs  in 
one  window.  If  they  were  working  on  two,  it  was  not  possible  to  tell  from  the 
trace  if  they  were  finishing  the  work  on  one  before  working  on  the  other  or  if 
they  were  being  worked  on  simultaneously.  The  videotape  records  clearly 
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indicate  these  features  of  problem  solving.  Rather,  the  rater  working  only 
from  the  computer  trace  had  to  make  inferences  about  what  had  been  worked 
on  in  the  K-map  window  based  on  subsequent  information  in  the  trace. 

These  software  features  led  to  the  discrepancies  between  the  raters. 

Implementation  Strategies.  Here  again,  lack  of  sufficient  detail  in  the 
information  gathered  in  the  computer  traces  resulted  in  the  raters  agreeing 
on  only  9  of  the  problem  A  implementations  and  on  only  8  of  18  problem  B 
implementations.2  Many  of  the  participants  worked  on  the  implementations 
for  both  outputs  in  the  same  window  and  the  computer  trace  did  not  indicate 
this.  Based  on  the  videotape-derived  strategies,  12  of  the  20  problem  A  and  15 
of  the  20  problem  B  implementations  were  parallel.  From  the  computer  traces 
only  3  problem  A  and  4  (of  18)  problem  B  implementations  appeared  to  be 
parallel.  The  lower  numbers  can  be  attributed  to  the  fact  that  this  computer 
trace  rater  had  no  information  on  how  the  connections  among  components 
were  being  established.  Thus,  there  was  a  lower  estimate  of  parallel 
implementation. 

Completed  Outputs.  Just  as  lack  of  sufficient  detail  in  the  computer 
traces  led  to  low  rater  agreement  on  the  implementation  strategies,  there  was 
low  agreement  on  the  number  of  outputs  implemented.  Raters  agreed  on  9 
problem  A  and  8  problem  B  solutions.  Based  on  the  videotape-derived 
information,  only  three  problem  A  and  two  problem  B  solutions  failed  to 
deal  with  both  outputs. 

In  addition  to  the  problems  with  the  lack  of  detail  in  the  Design 
Assistant  tool  traces,  information  about  the  purpose  for  various  actions  was 
not  captured.  However,  the  Design  Assistant  tool  was  adequate  for  purposes 
of  describing  the  overall  strategy  and  intermediate  steps  strategies.  The  results 
of  the  study  of  planning  (James,  1995)  indicated  that  this  information,  which 
is  reliably  derivable  from  the  computer  trace  generated  by  the  Design 
Assistant,  may  be  useful  in  distinguishing  among  levels  of  expertise. 

Effects  of  Planning  on  Digital  Circuit  Design 

To  further  investigate  the  effects  of  planning  during  simple  digital 
circuit  design,  the  next  study  (James,  1995)  examined  the  effects  of  anticipatory 
planning  on  the  problem-solving  processes  involved  in  solving  the  circuit 
design  problems.  Twenty  intermediate-skilled  electrical  engineering  students 
were  asked  to  think  aloud  while  solving  two  design  problems.  Before  solving 
the  second  problem,  participants  were  asked  to  create  a  detailed  global  plan  of 
their  problem  solving.  Subsequently,  these  plans  were  categorized  by  an 
expert  into  high,  medium,  or  low  expertise  categories.  The  contents  of  the 
plans  were  exhaustively  coded  into  three  categories  of  statements.  The 
problem-solving  protocols  were  examined  for  different  types  of  strategies  and 


2Details  of  the  implementation  steps,  such  as  the  connections  between 
gates,  were  not  recorded. 
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solution  quality. 

The  main  hypotheses  of  this  study  had  to  do  with  the  effects  of  asking 
participants  to  plan  their  solutions  prior  to  actually  beginning  the  design  task. 
It  was  expected  that  planning  would  lead  to  increased  quality  of  solution,  as 
indexed  by  optimization  attempts,  number  of  component  steps  attempted, 
and  correctness  of  the  final  solution.  The  planning  manipulation  did  not 
significantly  affect  any  of  the  measures  of  solution  quality.  The  reason  for  the 
lack  of  effects  due  to  planning  may  have  been  because  the  plans  were  not 
perceived  as  helpful.  Accordingly,  a  second  set  of  analyses  examined  whether 
participants'  perceived  helpfulness  of  the  planning  was  significantly  related  to 
the  same  three  measures  of  solution  quality  as  well  as  to  solution  strategy. 
However,  there  were  no  significant  effects  related  to  perceived  helpfulness  of 
the  plan. 

Although  there  were  no  significant  effects  of  the  planning 
manipulation,  there  were  significant  relationships  between  the  expert's  rating 
of  the  plans  and  (a)  the  contents  of  the  plans,  and  (b)  the  overall  problem¬ 
solving  strategy.  A  content  analysis  of  the  plans  indicated  that  participants 
with  more  Planning-Related  Statements  in  their  plans  were  more  likely  to 
receive  higher  plan  ratings.  Their  plans  included  more  action  statements, 
explanations,  and  evaluations  than  did  the  plans  that  received  medium  and 
low  ratings  from  the  expert. 

One  of  the  interesting  findings  of  the  study  concerns  the  overall 
problem  solving  strategies  used  by  the  participants  who  attained  higher  plan 
ratings.  Participants  who  attained  higher  plan  ratings  employed  Boolean 
equations  in  their  problem  solutions  more  frequently  than  the  other 
students.  Given  the  backgroimd  and  capabilities  of  the  participants,  it  seems 
reasonable  to  conclude  that  the  few  students  who  vmderstood  the  nature  of 
Boolean  expressions  and  were  willing  to  manipulate  them  directly  (for 
purposes  of  transformation  and  reduction)  had  a  more  complete 
understanding  of  the  domain,  especially  the  relation  between  logic  gates  and 
Boolean  expressions.  Most  of  the  other  students  appeared  to  apply  the 
component-step  procedures  mechanically  without  a  true  imderstanding  of 
what  they  were  achieving.  This  observation  is  further  supported  by  the 
number  of  errors  students  made  in  their  final  solutions,  and  their  lack  of 
ability  to  detect  and  correct  these  errors.  Both  tendencies  were  related  to  expert 
plan  rating. 

Contrary  to  general  expectations,  providing  an  opportunity  by  asking 
for  an  anticipatory  plan  did  not  increase  the  quality  of  the  participants' 
solutions.  Anticipatory  planning  is  hypothesized  to  be  most  helpful  in 
situations  where  the  participant  can  reduce  the  problems  to  well-structured, 
familiar  and  simple  forms  or  a  set  of  simpler  subcomponents  (Scholnick  & 
Friedman,  1987).  The  problems  in  the  present  study  were  intended  to  be  well- 
structured.  However,  approximately  half  the  errors  made  in  solving  these 
problems  (48%)  occurred  in  generating  the  truth  table.  An  additional  39% 
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occurred  in  the  K-map  component.  These  errors  resulted  in  participants 
trying  to  solve  rather  arbitrary,  unfamiliar,  and  complex  problems.  On  such 
problems,  anticipatory  planning  would  not  be  appropriate  nor  expected  to 
have  beneficial  effects  on  problem  solving  (Scholnick  &  Friedman,  1987).  The 
occurrence  of  errors  in  the  early  components  may  also  explain  why  plan 
rating  was  not  predictive  of  correctness  of  the  solution  nor  of  minimization 
and  optimization  measures. 

The  findings  of  the  planning  study  suggest  potentially  fruitful  areas  of 
future  research,  including  investigations  of  the  use  of  Boolean  equations  by 
more  expert  designers.  It  is  possible  that  the  effective  use  of  Boolean 
equations  is  a  benchmark  in  the  acquisition  of  expertise  in  this  domain. 
Results  from  this  study  indicated  that  the  global  plans  generated  by 
intermediate  participants  were  not  detailed  enough  to  describe  and  capture 
the  entire  solution  space.  It  might  also  be  interesting  to  determine  at  what 
point  in  the  acquisition  of  expertise  the  student  of  design  can  and  does 
formulate  a  detailed  global  problem-solving  plan  spontaneously,  and  at  what 
point  such  plans  can  be  formulated  when  explicitly  prompted.  Our 
experiences  suggest  that  providing  an  accurate  truth  table  may  well  facilitate 
studying  the  relation  between  planning  and  problem  solving. 

Differences  in  overall  and  intermediate  steps  solution  strategies  were 
significantly  related  to  the  expert's  plan  ratings.  TTiese  are  two  indices  that 
could  be  reliably  determined  from  the  computer  trace  generated  by  the  Design 
Assistant  tool.  Thus  it  seems  that  efforts  to  develop  computer-based, 
performance-oriented  design  process  assessments  is  a  fruitful  direction. 

Designing  Complex  Circuits 

One  of  the  difficulties  encountered  in  the  design  process  analysis  of  the 
combinational  logic  circuit  problems  was  their  "beginning"  nature.  This  focus 
was  necessary  to  pursue  the  expert  -  intermediate  contrastive  comparison  we 
were  interested  in.  Yet,  the  basic  nature  of  these  circuits  contributed  to 
difficulties  encoimtered  by  the  experts.  In  practice,  circuit  design  at  this  level 
has  become  automated.  Combinational  logic  circuits  are  often  elements  of 
more  complex  systems  that  experts  design.  They  are  "off  the  shelf" 
components  that  can  be  plugged  into  larger  designs  without  being  generated 
anew  each  time  they  are  needed.  Thus,  experts  rarely  design  these  kinds  of 
circuits  once  they  have  mastered  basic  and  intermediate  levels  of  circuit 
design.  Accordingly,  the  other  major  strand  of  work  in  this  project  examined 
the  design  process  for  very  complex  digital  circuits. 

TTie  complexity  in  engineering  design  can  be  attributed  to  the  fact  that 
mapping  from  a  specified  functionality  to  a  realizable  physical  structure  (the 
designed  artifact).  Complexity  arises  from 

the  specified  functionality.  Complex  functions  often  have  to  be  broken  down 
into  sub-functionalities  that  interact  (cf.  leak}/  modules  (Goel  &  Pirolli,  1991)). 
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Composing  the  individual  sub-functionality  implementations  becomes  a 
difficult  task. 

mediating  the  relation  between  function,  structure,  and  behavior.  Behavior 
acts  as  the  intermediary  between  function  and  structure.  A  designer 
hypothesizes  a  structure,  determines  its  behavior,  and  then  compares  this  to 
the  functionality  (expected  behavior).  This  process  is  non  trivial;  it  defines  a 
non-linear  iterative  framework  for  the  design  process. 

additional  criteria,  such  as  performance  parameters  and  aesthetics  that  may  be 
imposed  on  the  design  task. 

A  primary  goal  of  this  part  of  the  project  was  to  identify  the  differences 
in  the  processes  employed  by  experts  and  novices  in  dealing  with  complex 
design  problems  with  the  above  characteristics.  As  discussed  earlier,  the 
problem  with  combinational  circuit  design  was  its  routine  and  algorithmic 
nature.  Furthermore,  the  availability  of  automated  Computer  Assisted  Design 
(CAD)  tools  in  this  domain,  implies  that  experts  and  practitioners  are  rarely 
involved  in  the  details  of  combinational  circuit  design.  On  the  other  hand, 
current  digital  design  activity  primarily  focuses  on  systems,  such  as 
processors,  computer  subsystems,  memory  subsystems,  and  network 
controllers  that  include  elements  of  combinational  and  sequential  circuit 
design.  The  complexity  of  their  functional  descriptions  requires 
decomposition  into  subproblems  to  achieve  solutions.  In  addition,  their 
behavior  descriptions  are  dynamic,  i.e.,  their  behavior  at  a  certain  time  step  is 
dependent  on  the  state  of  the  system  at  the  previous  time  step,  making 
analysis  and  understanding  of  such  systems  a  more  difficult  task.  Our 
experience  with  the  simpler  combinational  circuit  suggested  that  shifting  the 
emphasis  to  complex  design  tasks  would 

•  allow  the  study  of  systematic  strategies  in  generating  design 
solutions,  and 

•  make  explicit  the  complex  and  multi-dimensional  nature  of  expert 
problem  solving  processes. 

In  studying  complex  circuit  design  activity,  it  became  clear  that  good  designers 
had  to  catch  flaws  and  resolve  discrepancies  early  in  the  design  process  to 
avoid  being  overwhelmed  by  the  magnitude  and  complexity  of  the  problem 
in  later  stages.  Hence,  we  were  interested  in  characterizing  the  processes 
involved  in  producing  the  final  design,  as  well  as  the  completeness  and 
correctness  of  the  final  design  itself. 

Characterizing  Complex  Design  Activity 

Complex  design  tasks  have  been  classified  into  three  broad  levels  of  design 
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activity  (Tong  and  Sriram,  1992):3 

•  Design  activities  at  the  function  level  study  the  specifications  of  a 
system  and  generate  functional  units  that  collectively  meet  the 
required  specifications. 

•  The  transition  level  includes  design  activities  that  look  to 
functional  units  and  decide  how  to  implement  them  as  artifacts 
(e.g.,  gates  and  higher-level  blocks  such  as  shift  registers  and 
counters).  Actions  at  this  level  may  also  guide  the  coordination  of 
component  artifacts  so  that  higher-level  functionality  is  satisfied. 

•  Implementation  level  activities  deal  with  the  actual  generation  of 
the  artifact.  In  complex  design  problems  this  often  happens  in 
stages;  for  example,  simulation,  testing,  and  evaluation  of  partial 
designs. 

Researchers  have  also  described  orthogonal  gradients  that  characterize 
the  cognitive  processes  of  the  designer,  independent  of  the  artifact  being 
designed  (Korpi,  Greeno  &  Jackson,  1991,  Goel  &  Pirolli,  1991,  Ullman  et  al., 
1988).  In  this  work,  we  have  found  two  descriptive  levels  of  reasoning 
sufficient:  the  strategy  and  unit  operator  levels  (for  details  see,  Biswas  et  al., 
1994).  In  keeping  with  our  interests  in  studying  planning  behavior  in  design 
problem  solving,  our  characterization  of  design  activity  was  focused  on  the 
strategy  level.  A  survey  of  past  and  present  design  research  indicated  a  set  of 
high-level  problem  solving  strategies  for  design. 

•  Decomposition  strategies  subdivide  a  complex  design  task  into 
smaller  sub-tasks  until  they  are  directly  solvable.  Examples  of  design 
systems  that  employ  decomposition  are  Hi-Rise  (Maher,  1988)  and 
R1  (McDermott,  1982). 

•  Transformation/Refinement  strategies  convert  initial  specifications 
into  a  final  design  solution  through  a  sequence  of  primitive 
operator  applications  in  some  formal  representation  scheme  (e.g., 
shape  grammars  (Mitchell  and  Stiney,  1978)). 

•  Case-based  strategies  assume  that  a  catalog  of  previous  design 
solutions  is  stored  in  memory;  a  new  design  problem  prompts  the 
designer  to  search  through  this  catalog,  and  select  one  or  more 
designs  that  best  match  the  characteristics  of  the  goal  design. 
Examples  of  case-based  approaches  to  design  include  the  Bogart 
system  (Mostow,  Barley,  and  Weinrich,  1989)  and  Struple  (Zhao  and 
Maher,  1988). 

We  hypothesized  the  presence  of  each  of  these  strategies  in  designers.  The 


3This  is  a  broader  and  richer  description  of  design  activity  than  the  overall, 
intermediate,  and  implementation  strategies  described  for  the  combinational  circuit 
problems. 
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possibility  of  multiple  strategies  in  a  single  design  solution  provided  a  much 
richer  framework  for  characterizing  the  reasoning  methods  of  designers  in 
complex  problem  solving. 

Design  can  be  looked  at  as  an  application  of  reasoning  processes  that 
start  at  the  function  level,  go  through  a  transition  level,  and  produce  an 
implementation-level  description  of  the  artifact.  Several  strategies  can  be 
applied  in  this  process.  The  directedness  with  which  these  strategies  were 
employed  and  coordinated  by  our  participants  was  used  as  the  basis  for 
assessing  a  designer's  expertise  relative  to  a  design  problem  or  class  of 
problems. 

A  Study  of  Complex  Circuit  Design 

Our  objective  in  this  study  was  to  correlate  plarming  behavior  and  the 
use  of  strategies  with  levels  of  expertise.  We  selected  a  complex  problem, 
designing  a  network  controller  to  coordinate  communications  between 
computers  that  are  linked  by  cable.  A  more  complete  specification  for  this 
problem  is  given  in  Table  4.  Our  experience  with  this  and  similar  problems 
suggested  numerous  activities  that  would  be  observed  across  a  range  of 
designers. 

We  administered  the  network  controller  problem  to  eleven 
participants.  Three  were  considered  experts:  S8,  a  faculty  member  at 
Vanderbilt  University  who  teaches  digital  design  and  had  designed  digital 
systems  for  industrial  applications,  S7,  who  works  in  the  communications 
technology  industry,  and  Sll,  a  graduate  student  with  extensive  CMOS  digital 
circuit  design  experience  in  industry.  Two  participants,  S4  and  SIO,  had 
related  industrial  experience,  but  neither  could  be  considered  to  be  experts  in 
complex  circuit  design.  Four  participants  (S2,  S5,  S6,  and  S9)  were  graduate 
students  with  little  or  no  professional  design  experience.  The  other  two 
participants  were  seniors  in  the  undergraduate  program  at  Vanderbilt 
university.  They  both  had  completed  a  senior  level  course  in  digital  circuit 
design.  However,  participant  S3  was  regarded  as  a  high-performing  student, 
whereas  SI  was  considered  to  be  average  in  academic  performance. 

Protocol  Analysis 

The  participants  were  given  a  hard  copy  of  the  problem  description  (see 
Table  4),  and  asked  to  develop  their  solution  with  pencil  and  paper.  They 
were  requested  to  think  aloud  as  they  developed  their  solution,  and  the  entire 
session  was  videotaped.  The  participants  could  use  any  material  (notes,  books, 
etc.)  that  they  felt  would  assist  them  in  generating  a  solution.  A  complete 
transcript  of  each  session  was  generated. 

Later  a  group  of  raters  (Biswas,  Glewwe,  Bhuva)  studied  the  tapes, 
transcripts,  and  the  participants'  written  output.  To  encode  strategic  activity, 
the  following  two-step  process  was  used. 
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•  In  Step  1,  we  encoded  the  implementation-level  blocks  created  by 
the  participants,  and  the  sequence  in  which  they  added,  deleted,  and 
modified  blocks  as  they  went  through  the  design  process.  This  trace 
recorded  the  designer's  activities  at  the  implementation  level.  Most 
implementation  level  blocks  originated  from  a  function-level 
description  and  retained  the  same  name  as  the  functional  block 
(e.g.,  serial-to-parallel  buffer).  Participants  usually  put  down  this 
name  when  they  drew  this  block  as  part  of  the  implementation  on 
paper,  or  they  verbally  expressed  the  name  of  the  block  as  they  drew 
it  on  paper. 

•  In  Step  2,  we  used  the  Step  1  trace  and  information  from  the  tapes 
and  transcripts  to  answer  the  set  of  questions  that  define  our 
framework  for  characterizing  design  activity  (see  Biswas  et  al.,  1994 
for  details).  This  provided  information  on  strategic  level  behavior 
of  participants  at  the  function,  transition,  and  implementation 
levels. 

The  trace  generated  in  Step  1,  in  addition  to  providing  information  for 
creating  Step  2,  also  helped  us  check  the  correctness  of  the  design.  If  the 
participant  had  errors  in  the  design,  the  trace  helped  us  understand  where 
and  how  a  participant  went  wrong.  The  analysis  was  performed  in  two  steps: 
(i)  use  the  Step  1  and  Step  2  traces  to  characterize  and  analyze  the  participants' 
solution  processes,  i.e.,  process  characteristics,  and  (ii)  use  the  Step  1  results  to 
analyze  and  evaluate  the  correctness  of  the  final  solution  generated  by  the 
participants,  i.e.,  product  characteristics.  Process  characteristics  were  studied  at 
the  function,  transition,  and  implementation  levels. 

Results  of  the  analyses 

Process  Characteristics.  Tables  5,  6,  and  7  summarize  the  individual 
behaviors  of  our  11  participants  along  the  function,  transition,  and 
implementation  levels,  respectively. 

Table  5  shows  the  evaluation  of  each  participant  at  the  function  level. 
For  this  analysis,  we  assumed  a  normative  solution,  consisting  of  7  functional 
blocks,  and  an  intermediate  form  (IF)  given  by  the  flowchart  in  Fig.  3.  This 
constituted  the  'gold  standard'  against  which  we  compared  participant  s' 
solutions.  We  were  interested  in  assessing  how  close  to  this  solution  each 
participant  came.  Because  participants  might  differ  in  the  number  and 
organization  of  functional  blocks,  we  developed  an  algorithm  for  coimting 
the  amount  of  'functional'  overlap.  If  a  designer  defined  a  fimctional  block 
that  covered  more  than  one  of  the  7  specified  blocks,  all  'covered'  blocks  in 
the  normative  solution  were  counted  as  considered.  If  the  designer  used 
finer-grain  blocks  than  the  ones  specified,  a  set  of  finer-grain  blocks  were 
counted  as 

a  larger  normative  block  if  the  designer  effectively  'implemented'  this  block 
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with  appropriate  links  between  the  finer  grained  blocks. 

In  addition.  Table  5  differentiates  between  the  number  of  functional 
blocks  initially  considered  (obtained  from  recorded  protocols),  and  the 
number  of  blocks 

that  were  represented  in  the  final  intermediate  form;  these  are  given  in 
columns  1  and  3,  respectively.  Other  features  used  for  assessment  are  the  type 
of  Intermediate  Form  (IF)  used,  details  specified  in  IF  (i.e.,  detail  in  specifying 
links  between  blocks),  and  verbal  evidence  for  the  presence  of  a  global  plan 
(i.e.,  some  explication  by  the  participants  of  the  steps  that  they  would  follow 
in  elaborating  the  design).  Some  of  the  participants  did  not  use  an  IF  and  they 
were  evaluated  based  on  the  rest  of  the  design  features.  The  use  of  higher 
number  of  components,  greater  detail  in  IF,  higher  number  of  components  in 
IF,  and  the  presence  of  a  global  plan  implied  higher  levels  of  expertise. 
However,  in  assigning  an  overall  rating  of  observed  competence  at  the 
function  level  (column  6),  we  focused  on  the  coverage  and  detail  of  the  IF. 

Table  6  shows  the  evaluations  of  each  participant  at  the  transition 
level.  The  features  used  were  the  details  expressed  in  decomposition  and  the 
handling  of  interactions  in  this  level.  In  general,  the  more  fine-grained  the 
decomposition  and  the  better  the  expression  of  interactions  between  modules 
expressed  in  the  form  of  signals,  then  the  greater  the  ease  with  which  design 
should  proceed  through  the  implementation.  The  better  the  participants 
performed  along  these  two  dimensions  the  greater  was  their  rated  level  of 
expertise.  Column  4  in  the  table  provides  overall  ratings. 

Table  7  shows  the  evaluations  of  each  participant  at  the 
implementation  level.  The  features  used  for  assessment  are  the  use  of  higher- 
level  components,  use  of  custom  components,  t)^es  of  interaction  checks 
made,  and  the  strategy  employed  in  problem  resolution.  This  table  does  not 
give  an  overall  rating,  but  we  determined  that  the  use  of  high  level  and 
custom  components  demonstrated  greater  expertise,  as  did  global  versus  local 
interaction  checks.  In  addition,  most  participants  used  patching  to  correct  for 
interactions. 

Product  Characteristics.  We  also  characterized  participants  in  terms  of 
the  completeness  and  correctness  of  their  final  designs.  Table  8  gives  a 
qualitative  score  for  the  completeness  of  each  participant's  final 
implementation  (i.e.,  what  proportion  of  the  specified  fimctionality  did  the 
final  design  cover)  and  a  score  for  correctness  (i.e.,  the  proportion  of  specified 
subfimctionalities  correctly  implemented  in  the  final  solution).  Table  78  also 
classifies  participants  on  a  four-point  scale,  on  the  basis  of  the  combined 
completeness  and  correctness  ratings.  Level  1  scores  reflected  minimal 
completeness  and  correctness  in  final  implementation.  Level  2  scores 
reflected  medium  completeness  and  correctness.  Level  3  participants  scored 
high  on  one  dimension  and  medium  on  the  other.  One  participant  was 
classified  at  Level  4,  and  scored  high  on  both  dimensions  of  completeness  and 
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correctness.  We  left  participant  7  unclassified,  because  he  took  a  radically 
different  approach  to  the  controller  design  than  other  participants.  P7 
employed  a  programmable  logic  array  in  solving  the  problem,  and  was 
stopped  early  by  the  experimenter  prior  to  full  implementation. 

Integrating  Process  and  Product  Characteristics.  The  product  rating  and 
the  previously-described  process  characteristics  are  generally  consistent  with 
one  another.  An  exception  to  this  was  P3  who  performed  well  at  the  function 
and  transition  levels.  This  exception  would  have  gone  unnoticed  had  we 
only  looked  at  product  scores.  We  were  concerned  that  other  important 
differences  in  design  activity  would  be  masked  if  we  focused  solely  on  the 
product  measure.  We  developed  a  more  integrated  rating  scheme  consisting 
of  four  levels  as  follows. 

•  Novices  showed  little  proficiency  in  complex  design.  Novices 
scored  low  in  completeness  and  correctness,  and  had  low  fimctional 
and  transitional  scores  as  well. 

•  Average  designers  scored  medium  for  completeness  and  correctness 
and  demonstrated  average  performance  at  the  functional  and 
transitional  level,  which  is  indicative  of  some  imderstanding  of  the 
problem  and  a  strategy-level  implementations. 

•  Above  average  designers  demonstrated  good  understanding  of 
complex  design  in  formulating  global  plans  and  applying  strategies, 
but  did  not  perform  as  well  as  experts  in  decomposition  and 
handling  of  interactions  at  the  transition  level.  Their  average 
completeness  and  correctness  was  above  medium. 

•  The  expert  designer  had  a  very  good  understanding  of  design.  This 
individual  scored  high  on  both  dimensions  of  completeness  and 
correctness,  and  demonstrated  mastery  at  the  fimctional  and 
transitional  levels.  The  key  to  the  expert's  success  can  to  a  large 
extent  be  attributed  to  his  ability  to  handle  complexities  at  the 
function  and  transition-levels  of  design. 

These  categories  were  exemplified  in  the  descriptions  of  the  design 
problem  solving  behavior  of  our  participants  (see  Biswas  et  al.,  1994  for 
details),  and  provide  a  strong  link  between  process  characteristics  and  the 
quality  of  the  final  product  generated  by  a  designer. 

Implications  for  Assessment  of  Expertise 

This  study  made  it  clear  that  there  was  important  information  to  be 
gained  by  looking  at  how  designers  went  about  producing  a  final  design.  For 
example,  having  only  looked  at  products,  i.e.,  the  resultant  designs,  we 
would  not  have  known  that  participants  9  and  10  used  a  primitive  form  of 
case-based  reasoning  and  did  not  use  functional  decomposition.  Likewise,  P3's 
performance  was  due  to  difficulties  at  the  implementation  level,  while  her 
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performances  at  the  function  and  transition  levels  were  quite  good.  We 
would  also  not  have  known  that  P4  attempted  to  transfer  his  expertise  in 
analog  circuit  design  to  the  digital  case.  Furthermore,  Pll's  behavior 
suggested  the  importance  of  early  error  checking  and  evaluation  at  the 
functional  level  in  obtaining  a  good  design.  The  information  gained  from  the 
process  level  descriptions  provides  a  basis  for  comparisons  with  other 
domains  in  which  expertise  has  been  described.  In  addition,  a  comparison  of 
the  processes  used  by  the  most  expert  participant  in  our  group  with  the 
processes  of  the  less  expert  participants  provides  a  basis  for  instructional 
interventions. 

Design  Tool  for  Complex  Circuit  Design 

Assuming  that  one  is  interested  in  the  richer  characterization  that 
results  from  considering  both  process  and  product  descriptions  in  digital 
circuit  design,  it  is  important  to  make  the  collection  of  process  data  more 
tractable  than  through  the  analysis  of  think-aloud  verbal  protocols.  Towards 
this  end,  we  built  a  prototype  design  tool  to  assist  participants  in  function  and 
intermediate-level  design.  The  design  tool  has  the  following  components: 

•  problem-description  browsing  tool 

•  high-level  formalization  and  simulation  tool 

•  circuit-diagram  drawing  tool,  and 

•  circuit-simulator  tool 

Preliminary  testing  conducted  on  participants  as  well  as  the  analysis  of 
the  complex  problem  described  above  made  it  clear  that  to  perform  more 
detailed  experiments  (cf.  James,  1995)  would  require  that  the  system  provide 
the  capability  to  generate  a  number  of  intermediate  forms  (IFs)  and  the  ability 
to  simulate  these  IFs  to  check  the  behavior  of  the  designed  system.  However, 
the  termination  of  the  project  did  not  allow  us  to  pursue  development  of  the 
design  assistant  tool  any  further.  Future  work  in  this  area  should  be  directed 
toward  the  building  of  fimction-level  design  tools  and  the  capture  of  design 
activity  traces  at  this  level  of  design. 

Conclusions 

The  diagnosis  of  expertise  in  a  complex  design  task  first  requires  a 
sufficient  understanding  of  sources  of  expertise-related  differences.  The 
studies  conducted  imder  the  auspices  of  this  project  suggest  some  preliminary 
characterizations.  In  addition,  the  work  demonstrates  the  feasibility  as  well  as 
current  limitations  of  using  computerized  automated  diagnosis  systems. 

The  study  of  problem  solving  behavior  for  the  simpler,  combinational 
circuit  provided  a  characterization  of  expert-novice  differences. 
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•  Intermediate  and  novice  problem  solvers  exhibited  a  large  amount 
of  "step-by-step"  local  planning.  These  often  occurred  at  transitions 
points  from  one  algorithmic  component  to  the  next.  This  pattern 
seemed  to  indicate  that  the  intermediate  and  novice  problem 
solvers  were  somewhat  mechanically  implementing  an  algorithmic 
solution  without  reflecting  on  the  consequences  and  results  they 
were  obtaining  as  they  solved  the  problem.  In  contrast,  the  experts 
appeared  to  be  proceeding  with  fewer  difficulties  and  to  have  a  in 
mind  a  plan  for  what  they  would  do  next.  Although  only  one  expert 
verbalized  a  global  plan,the  other  expert  showed  little  of  the 
indecision  shown  by  the  intermediates  when  they  completed  a 
component. 

•  To  facilitate  improved  problem  solving  by  intermediates,  a  second 
study  required  them  to  generate  and  verbalize  a  global  plan  prior  to 
beginning  solution.  The  forced  generation  of  a  global  plan  did  not 
impact  any  indices  of  problem  solving.  However,  expertness  of  the 
plan  generated  was  predictive  of  overall  problem  solving  strategy. 
The  primary  differentiation  was  that  more  expert  planners  included 
Boolean  expressions  in  their  solutions  more  frequently  than  less 
expert  planners. 

We  did  not  obtain  the  magnitude  of  differentiation  among  experts  and 
intermediates  that  we  had  expected.  We  attribute  this  to  two  related  factors. 
Combinational  circuit  design  is  an  extremely  routine  task.  Consequently,  the 
experts  were  not  in  the  practice  of  designing  these  "by  hand"  but  relied  on 
automated  tools  to  generate  these  circuits.  Second,  errors  made  early  in  the 
solution  process  made  it  quite  difficult  for  all  participants  to  discern  the 
relevant  patterns.  Had  the  experts  worked  on  problems  that  were  closer  to 
their  daily  job  activities,  we  suspect  differences  between  them  and 
intermediates  would  have  been  greatly  magnified. 

A  second  goal  of  our  work  with  the  combinational  circuit  design  task 
was  to  implement  an  automated  system  for  administering  and  assessing 
students.  A  major  issue  was  whether  computer  traces  of  the  design  process 
would  be  sufficient  to  distinguish  among  levels  of  expertise  in  design.  The 
computer  traces  generated  from  the  Design  Assistant  tool  contained 
information  sufficient  to  reliably  discern  designers'  overall  and  intermediate 
steps  strategies.  These  were  the  two  descriptors  of  problem  solving 
performance  that  were  significantly  related  to  expertness  of  the  generated 
plan.  We  also  realized  that  the  design  tool  was  not  collecting  sufficient  design 
activity  information  to  determine  a  number  of  other  important  elements  of 
the  design  solution  process.  Several  changes  that  would  ameliorate  this 
problem  could  be  rather  easily  implemented  in  a  "next"  version  of  the  tool. 

The  more  complex  sequential  circuit  design  task  proved  to  be  quite 
successful  in  differentiating  expert,  intermediate,  and  novice  behavior.  This 
was  achieved  by  analysis  of  the  high  level  strategies  the  participants  employed 
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at  the  ftmction,  transition,  and  implementation  levels  of  design.  A  number  of 
interesting  points  will  help  characterize  assessment  of  expert-novice 
differences  in  the  future. 

•  Experts  tend  to  focus  more  on  function-level  design  in 
decomposing  a  complex  system  into  sub  functionalities,  and 
defining  the  interactions  between  the  different  functional  blocks. 
This  guides  design  at  the  transition  and  implementation  levels. 
Intermediates  and  novices,  in  particular,  do  not  seem  to  put  in  as 
much  effort  at  the  function  level,  which  results  in  their  being 
overwhelmed  by  the  complexity  of  the  problem  at  the  transition 
and  implementation  levels.  The  end  result  is  erroneous  and/or 
suboptimal  design  solutions. 

•  Expert-novice  assessment  is  best  conducted  by  studying  process  data 
as  opposed  to  product  data  (i.e.,  just  the  quality  of  a  participant's 
final  solutions).  Furthermore,  analysis  of  process  data  can  be 
effectively  conducted  by  working  at  strategy-level  operations  (i.e., 
planning  behavior)  as  opposed  to  more  detailed  unit-process  level 
behavior. 

•  It  is  hard  to  characterize  expertise:  expert-level  behavior  manifests 
itself  in  multiple  ways  through  the  use  of  multiple  combinations  of 
basic  strategies.  The  one  common  thread  we  observed  in 
characterizing  experts  was  their  systematic  attempts  and  success  at 
tackling  the  harder  problems  early  in  the  design  process,  making  the 
error  handling  and  optimization  tasks  at  the  implementation  level 
a  much  easier  task. 

Comparing  the  framework  generated  and  the  results  produced  from 
the  two  sets  of  studies,  we  can  claim  that  we  were  able  to  define  a  sufficiently 
complete  framework  for  characterizing  both  process  and  product  aspects  of 
digital  circuit  design.  Our  comparative  studies  with  the  computer-based 
assessment  tool  for  combinational  circuits  indicated  a  number  of  areas  where 
more  detailed  information  was  required  in  the  traces  to  ensure  that  reliable 
conclusions  could  be  made  about  design  activity.  For  the  more  complex 
problems,  the  design  of  computer-based  tools  becomes  much  more  difficult, 
because  of  the  multiple  representations  that  designers  employ  in  fxmctional- 
and  intermediate-level  designs.  Furthermore,  it  is  still  an  open  question  as  to 
whether  the  strategic  behavior  of  designers  can  be  derived  from  computer 
traces  of  their  design  activity. 
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Table  1:  Combinational  circuit  design  problems 

Problem  A.  The  Subtractor  Problem 

You  are  to  design  a  1-bit  full  subtractor.  Your  circuit  will  accept  one  bit 
of  each  operand  and  an  input  borrow  bit  and  produce  a  result  bit  and  an 
output  borrow. 

In  solving  this  problem  keep  in  mind  that  we  are  not  interested  in  how 
much  time  it  will  take  you.  We  are  interested  in  seeing  a  good  design  and  the 
steps  leading  you  there.  Assume  you  will  have  to  make  a  product  out  of  this 
design.  The  circuit  will  have  to  be  implemented  in  a  CMOS  chip.  Your  criteria 
for  "good  design"  should  be  (in  order  of  importance): 

1.  Minimum  cost,  that  is,  minimum  number  of  gates  used.  Keep  in 
mind  that  some  CMOS  gates  are  more  complicated  than  others.  The 
goal  is  thus  actually  to  minimize  the  area  of  the  chip's  layout. 

2.  Testability 

Problem  B.  The  Tob  Skill  Problem 

A  certain  placement  service  evaluates  people  based  on  their  possession 
(or  lack)  of  four  basic  skills:  typing,  basic  computer  use,  friendliness,  and  god 
phone  voice.  Each  applicant  is  evaluated  for  Type  A  and  Type  B  jobs.  Being 
suitable  for  one  or  both  types  of  jobs  results  in  being  added  to  their  files  with 
the  t5pes  of  jobs  the  applicant  is  suitable  for.  If  an  applicant  is  suitable  for 
neither  job  they  are  rejected.  Type  A  job  skills  include  applicants  who:  carmot 
type,  but  can  use  a  computer,  and  have  a  good  phone  voice;  are  friendly,  have 
a  good  phone  voice,  and  can  type;  are  friendly,  can  use  a  computer,  but  can't 
type;  can't  use  a  computer,  don't  have  a  good  phone  voice,  aren't  friendly,  and 
may  or  may  not  be  able  to  type. 

T)pe  B  job  skills  include  applicants  who:  can't  type,  and  don't  have  a  good 
phone  voice,  but  can  use  a  computer;  can  type,  have  a  poor  phone  voice,  and 
are  friendly;  are  friendly,  but  can't  type,  and  don't  have  a  good  phone  voice; 
can't  use  a  computer,  aren't  friendly,  but  do  have  a  good  phone  voice. 

As  a  designer,  create  a  logic  circuits  that  outputs  A  and  B  from  four  inputs  w, 
X,  y,  and  z. 

In  solving  this  problem  keep  in  mind  that  we  are  not  interested  in  how 
much  time  it  will  take  you.  We  are  interested  in  seeing  a  good  design  and  the 
steps  leading  you  there.  Assume  you  will  have  to  make  a  product  out  of  this 
design.  The  circuit  will  have  to  be  implemented  in  a  CMOS  chip.  Your  criteria 
for  "good  design"  should  be  (in  order  of  importance): 

1.  Minimum  cost,  that  is,  minimum  number  of  gates  used.  Keep  in 
mind  that  some  CMOS  gates  are  more  complicated  than  others.  The 
goal  is  thus  actually  to  minimize  the  area  of  the  chip's  layout. 

2.  Testability 
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Table  2:  Example  global  plan 

Let  me  demonstrate  for  you  what  I  mean  by  an  overall  plan  for  the  task  of 
preparing  a  shopping  trip. 

"Well,  let's  see.  I  have  to  buy  food  for  next  week.  First  I'll  have  to 
figure  out  how  many  days  next  week  I'll  be  eating  dinner  home 
and  how  many  times  I'll  be  packing  a  lunch  for  work.  I  don 't 
want  to  get  too  much  food.  I  should  check  the  refrigerator  and 
the  cupboards  to  make  sure  that  I  don't  buy  something  that  I 
already  have.  Is  there  a  special  occasion  coming  up  next  week 
which  would  require  me  to  make  something  specific?  If  so,  then  I 
need  to  get  the  ingredients  for  it.  I  need  to  think  about  cost 
savings.  Do  I  have  any  relevant  coupons?  Which  store  should  I 
go  to?  If  I'm  not  in  a  hurry  I  can  go  to  MegaMarket.  If  I'm  in  a 
hurry  it  is  too  far  away.  Kroger's  is  closer  but  more  expensive." 

So  by  an  overall  plan,  I  mean  laying  out  things  that  need  to  be  thought  about, 
how  to  accomplish  them  and  contingencies  that  need  to  be  considered. 
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Table  3:  Overall  solution  strategies  based  on  video  trace 


Overall  Strategy 

Problem  A 

Problem  B 

Textbook 

12  (60%) 

11  (55%) 

Textbook  without  Boolean 

6  (30%) 

6  (30%) 

Textbook  without  Grouping 

2  (10%) 

2  (10%) 

Textbook  without  K-map 

0 

1  (5%) 
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Table  4:  Problem  description  for  complex  circuit  design  study 

A  small  company  selling  networking  supplies  is  experiencing  problems 
with  one  of  their  IC  suppliers  and  decides  to  design  their  own  network 
controller.  The  type  of  network  the  company  sells  is  a  proprietary  token  ring 
architecture.  The  token  ring  controller  will  be  manufactured  as  a  CMOS  chip 
by  an  external  firm,  using  2  micron  technology.  The  network  normally 
connects  5  to  100  computers,  with  each  computer  containing  one  network 
controller.  You  are  asked  to  design  the  sender  part  of  the  controller.  The  final 
design  should  be  a  gate  level  design  specification.  You  are  free  to  use  higher 
level  modules  like  shift  registers,  just  specify  the  implementation  of  one  bit 
of  the  register  at  the  gate  level. 

The  computers  are  connected  in  a  ring  topology.  Each  computer  has  a  unique 
7-bit  address.  A  frame  of  data  is  of  variable  length  and  consists  of  the 
following  elements: 

1.  an  8-bit  token,  indicating  the  head  of  a  frame  (1000  0000) 

2.  a  1-bit  busy  signal,  indicating  whether  frame  is  full  or  empty  (1  - 
full,  0  -  empty) 

3.  a  7-bit  address  identifying  the  intended  receiver 

4.  a  1-bit  signal  acknowledging  reception  by  the  receiver  (1  -  accepter, 

0  -  other) 

5.  a  7-bit  address  identifying  the  sender 

6.  an  N-byte  data  packet 


The  controller  has  the  following  lines 

1.  (RI)  Ring  in  (input) 

2.  (RO)  Ring  out  (output) 

3.  (DI)  Data  in  (input) 

4.  (DF)  Data  finished  (input) 

5.  (FD)  Fetch  data  (output) 

6.  (CL)  Clock  (input) 


Connected  to  the  ring  (receiving  data) 
Connected  to  the  ring  (sending  data) 
Connected  to  the  system;  Input  for 
data  to  be  sent 
System  has  no  more  data 
Controller  needs  next  data  bit  from 
system 

Synchronous  signal  for  all  computers 


When  the  controller  receives  the  token,  it  checks  the  next  bit  to  see  if 
the  following  frame  contains  data.  If  it  does,  the  controller  does  nothing 
(receiver  part  checks  to  see  if  data  is  intended  for  this  system,  and  initiates 
reception).  If  the  frame  is  empty,  the  local  system  is  allowed  to  send  data  if  it 
has  any  available  (DF  =  0).  In  that  case  the  next  7  bits  are  set  to  the  address  of 
the  intended  receiver.  The  following  bit  is  set  to  0  (acknowledge  signal),  and 
the  next  7  bits  are  set  to  the  address  of  the  sender.  Finally  data  transmission 
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begins.  When  no  more  data  is  available  (DF  =  1)  transmission  stops.  While 
the  system  is  sending  it  keeps  watching  for  the  return  of  the  token.  When  it 
receives  a  busy  bit  following  return  of  the  token  (busy  bit  generated  by  itself)  it 
HAS  TO  stop  sending  and  reverse  the  busy  bit  to  0.  Generation  of  sender  and 
receiver  addresses  all  happens  externally  on  the  Data  In  (DI)  line.  NOTE:  You 
are  responsible  for  two  outputs  (Ring  Out  and  Fetch  Data)  derived  from  four 
inputs  (Ring  In,  Data  In,  Data  Finished,  and  Clock);  see  the  figure  on  the 
previous  page  for  explanations  of  these  signals. 

Additional  Constraints: 

•  Minimum  buffer  size  per  controller:  8  bits 

•  Maximum  buffer  size  per  controller:  16  bits 

•  Speed  of  operation:  10  Mbps 

Please  do  not  worry  about  the  following  aspects: 

•  generation  and  synchronization  of  clock  signals 

•  power  down  situations  in  any  of  the  computers  on  the  ring 

•  conflicts  between  data  and  marker  bit  patterns 

•  reception  of  packets 

•  loss  of  token 

•  no  acknowledgement  from  receiver 

•  limited  buffer  capacity  of  the  net 
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Table  5:  Evaluation  of  function  levels 


Partici¬ 

pant 

Number  of 
Components 

IF 

Number  in 
IF 

Detail  in 
IF 

Global 

Plan 

Total 

Score 

PI 

3 

none 

n/a 

na/ 

No 

1 

P2 

3 

7 

Low 

No 

2 

P3 

6 

Flowchart 

6 

High 

No 

4 

P4 

5 

none 

n/a 

n/a 

No 

1 

P5 

3 

Flowchart 

3 

Low 

No 

2 

P6 

6 

Flowchart 

4 

Medium 

No 

2 

P7 

- 

Data  Flow  Diagram 

4 

P8 

6 

Informal  State  Machine 

5 

P9 

5 

none 

n/a 

n/a 

No 

1 

PIO 

7 

none 

n/a 

n/a 

Yes 

1 

Pll 

7 

State  Machine 

6 

High 

Yes 

4 

aRatings  were  assigned  according  to  the  following  criteria:  (1)  No  IF;  2  =  low 
or  medium  coverage;  low  or  medium  detail;  3  =  medium  coverage;  medium 
detail;  and  4  =  high  coverage;  high  detail. 
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Table  6:  Evaluation  of  transition  levela 


■ 

Core  Component 

Detail  in 
Decomposition 

Use  of  Signals 

Total 

Score 

PI 

High 

Low 

2 

P2 

Low 

Low 

1 

P3 

Abstract  Controller 

High 

Medium 

3 

P4 

Shift  Register 

Medium 

Medium 

2 

P5 

System  State  Flop 

High 

Medium 

3 

P6 

Shift  Register 

High 

Low 

2 

P7 

ROM  Sequencer 

- 

- 

- 

P8 

Shift  Register 

Medium 

Medium 

2 

Shift  Register 

Medium 

Low 

2 

Shift  Register 

Medium 

High 

3 

High  Level  Blocks 

High 

High 

4 

^Ratings  were  assigned  in  the  following  manner:  1  =  Low  decomposition;  low 
use  of  signals;  2  =  Medium  decomposition;  less  than  medium  use  of  signals;  3 
=  High  decomposition  and/or  high  use  of  signals,  but  not  both;  and  4  =  High 
decomposition  and  high  use  of  signals. 
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Table  7:  Implementational  level 


Parti¬ 

cipant 

Strategy 

High-Level 

Components 

Custom 

Components 

Interaction 

Checks 

Problem 

Resolution 

PI 

Iterative  Addition 

Yes 

No 

Local,  Some 
Global 

Abandoned 

Flawed  &  Correct 
Parts 

P2 

Iterative  Addition  & 
Refinement 

Yes 

Yes 

Local  (very 
little) 

Immediate  Patch 

P3 

n/a 

Yes 

Yes 

none 

n/a 

P4 

Iterative  Addition  & 
Refinement 

Yes 

No 

Local 

Immediate  Patch 

P5 

Iterative  Addition  & 
Refinement 

Yes 

No 

Global 

Immediate  Patch 

P6 

Iterative  Addition 

Yes 

No 

Local 

Immediate  Patch 

P7 

Complete  Block 

Layout 

Yes 

No 

none 

n/a 

P8 

Global  Layout; 

Iterative  Refinement 

Yes 

Yes 

Local  w/  a  Final 
Global 

Deferment 

P9 

Iterative  Refinement 
&  Addition 

Yes 

Yes 

Local 

Immediate  Patch 

PIO 

Iterative  Refinement 
&  Addition 

Yes 

Yes 

Local 

Immediate  Patch 

Pll 

Iterative  Refinement 
on  Signal/Blocks 

Yes 

Yes 

At  Block  Level 

Immediate  Patch 
(One  Deferment) 
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Table  8:  Overall  evaluation 


Circuit  Design  Assessment 

31 


reads  problem 
1 


I 


states  plan 


I 


defines  inputs/outputs 
3 


truth  table 


16,18,20 


It 


17,19 


verifies  sim.  output 
21 


I 


evaluate  -  accept 


Figure  1:  Example  of  serial  flow  diagram 


Circuit  Design  Assessment 
32 


reads  problem 

states  plan 

■ ; 

defines  inputs /outputs 

I 

truth  table 


simulate 

verify  sim.  output 
I  13 

evaluate  -  accept 


Figure  2:  Example  of  parallel  flow  diagram 
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